Signal processing apparatus, signal processing method, and program

ABSTRACT

A signal processing apparatus including a first position calculation unit that calculates a three-dimensional position of a target on a first coordinate system from a stereo image captured by a stereo camera, a second position calculation unit that calculates a three-dimensional position of the target on a second coordinate system from a sensor signal of a sensor capable of obtaining position information of at least one of a lateral direction and a longitudinal direction and position information of a depth direction, a correspondence detection unit that detects a correspondence relationship between the target on the first coordinate system and the target on the second coordinate system, and a positional relationship information estimating unit that estimates positional relationship information of the first coordinate system and the second coordinate system on the basis of the detected correspondence relationship.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit under 35 U.S.C. § 120 as acontinuation application of U.S. application Ser. No. 15/762,136, filedon Mar. 22, 2018, now U.S. Pat. No. 10,908,257, which claims the benefitunder 35 U.S.C. § 371 as a U.S. National Stage Entry of InternationalApplication No. PCT/JP2016/077397, filed in the Japanese Patent Officeas a Receiving Office on Sep. 16, 2016, entitled “SIGNAL PROCESSINGAPPARATUS, SIGNAL PROCESSING METHOD, AND PROGRAM”, which claims priorityto Japanese Patent Application Number JP2015-194134, filed in theJapanese Patent Office on Sep. 30, 2015, each of which applications ishereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present technology relates to a signal processing apparatus, asignal processing method, and a program, and particularly relates to asignal processing apparatus, a signal processing method, and a programthat enable calibration with high precision.

BACKGROUND ART

In recent years, more and more vehicles such as automobiles haveintroduced collision avoidance systems. The collision avoidance systemsavoid collisions by detecting a car or pedestrian in front andautomatically performing brake control or the like.

An object such as a car or pedestrian in front is detected through imagerecognition of an image captured by a stereo camera or using radarinformation provided by a millimeter-wave radar, a laser radar or thelike. Further, the development of object detection systems using both astereo camera and a radar, which is called sensor fusion, is alsoadvancing.

Sensor fusion needs calibration of the coordinate system of the stereocamera and the coordinate system of the radar to match an objectdetected by the stereo camera and an object detected by the radar. Forexample, PTL 1 proposes a method of performing calibration for sensorfusion by using a board (reflective board) dedicated to calibration.

CITATION LIST Patent Literature

[PTL 1]

JP 2007-218738A

SUMMARY Technical Problem

However, since the detection points are limited to where thecalibration-dedicated board is arranged, the calibration method usingthe calibration-dedicated board as in PTL 1 has a limitation incalibration precision.

The present technology has been made in view of the foregoingcircumstances and enables calibration with high precision.

Solution to Problem

A signal processing apparatus according to an aspect of the presenttechnology includes a first position calculation unit that calculates athree-dimensional position of a target on a first coordinate system froma stereo image captured by a stereo camera, a second positioncalculation unit that calculates a three-dimensional position of thetarget on a second coordinate system from a sensor signal of a sensorcapable of obtaining position information of at least one of a lateraldirection and a longitudinal direction and position information of adepth direction, a correspondence detection unit that detects acorrespondence relationship between the target on the first coordinatesystem and the target on the second coordinate system, and a positionalrelationship information estimating unit that estimates positionalrelationship information of the first coordinate system and the secondcoordinate system on the basis of the detected correspondencerelationship.

A signal processing method according to an aspect of the presenttechnology includes the steps of calculating a three-dimensionalposition of a target on a first coordinate system from a stereo imagecaptured by a stereo camera, calculating a three-dimensional position ofthe target on a second coordinate system from a sensor signal of asensor capable of obtaining position information of at least one of alateral direction and a longitudinal direction and position informationof a depth direction, detecting a correspondence relationship betweenthe target on the first coordinate system and the target on the secondcoordinate system, and estimating positional relationship information ofthe first coordinate system and the second coordinate system on thebasis of the detected correspondence relationship.

A program according to an aspect of the present technology causes acomputer to execute a process including the steps of calculating athree-dimensional position of a target on a first coordinate system froma stereo image captured by a stereo camera, calculating athree-dimensional position of the target on a second coordinate systemfrom a sensor signal of a sensor capable of obtaining positioninformation of at least one of a lateral direction and a longitudinaldirection and position information of a depth direction, detecting acorrespondence relationship between the target on the first coordinatesystem and the target on the second coordinate system, and estimatingpositional relationship information of the first coordinate system andthe second coordinate system on the basis of the detected correspondencerelationship.

In an aspect of the present technology, a three-dimensional position ofa target on a first coordinate system from a stereo image captured by astereo camera is calculated, a three-dimensional position of the targeton a second coordinate system from a sensor signal of a sensor capableof obtaining position information of at least one of a lateral directionand a longitudinal direction and position information of a depthdirection is calculated, a correspondence relationship between thetarget on the first coordinate system and the target on the secondcoordinate system is detected, and positional relationship informationof the first coordinate system and the second coordinate system on thebasis of the detected correspondence relationship is estimated.

Note that the program can be provided by being transmitted via atransmission medium or by being recorded in a recording medium.

The signal processing apparatus may be an independent apparatus or maybe an internal block constituting one apparatus.

Advantageous Effect of Invention

An aspect of the present technology enables calibration with highprecision.

Note that the effect described herein is not necessarily limitative, andany of the effects described in the present disclosure may be exhibited.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary configuration of anobject detection system to which the present technology is applied.

FIG. 2 is a view illustrating an exemplary target used for apre-shipment calibration process.

FIG. 3 is a diagram illustrating an exemplary arrangement of targets inthe pre-shipment calibration process.

FIG. 4 is a diagram illustrating the exemplary arrangement of thetargets in the pre-shipment calibration process.

FIG. 5 is a diagram for describing a target detection unit and athree-dimensional position calculating unit.

FIG. 6 is a diagram for describing a target detection unit, a disparityestimation unit, and a three-dimensional position calculating unit.

FIG. 7 is a diagram for describing a correspondence detection unit.

FIG. 8 is a diagram for describing a position and attitude estimationunit.

FIG. 9 is a diagram for describing a first correspondence detectionprocess.

FIG. 10 is a diagram for describing the first correspondence detectionprocess.

FIG. 11 is a diagram for describing the first correspondence detectionprocess.

FIG. 12 is a diagram for describing the first correspondence detectionprocess.

FIG. 13 is a diagram for describing a second correspondence detectionprocess.

FIG. 14 is a block diagram illustrating a detailed exemplaryconfiguration of the target detection unit in which a calibrationprocess during operation is executed.

FIG. 15 is a block diagram illustrating a detailed exemplaryconfiguration of the target detection unit in which the calibrationprocess during operation is executed.

FIG. 16 is a diagram for specifically describing the calibration processduring operation.

FIG. 17 is a diagram for specifically describing the calibration processduring operation.

FIG. 18 is a diagram for specifically describing the calibration processduring operation.

FIG. 19 is a diagram for specifically describing the calibration processduring operation.

FIG. 20 is a diagram for specifically describing the calibration processduring operation.

FIG. 21 is a diagram for specifically describing the calibration processduring operation.

FIG. 22 is a flowchart for describing a calibration process.

FIG. 23 is a flowchart for describing a calibration process at shipmentin FIG. 22 .

FIG. 24 is a flowchart for describing the calibration process duringoperation in FIG. 22 .

FIG. 25 is a diagram illustrating other examples of the targets.

FIG. 26 is a diagram illustrating an example where the targets areballs.

FIG. 27 is a diagram for describing a method of detecting a targetposition where the target is a pole-like object.

FIG. 28 is a block diagram illustrating an exemplary configuration of anembodiment of a computer to which the present technology is applied.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a mode for carrying out the present technology (hereinafterreferred to as an embodiment) will be described. Note that thedescription will be made in the following order.

1. Exemplary Configuration of Object Detection System

2. Detailed Description of Correspondence Detection Process

3. Calibration Process During Operation

4. Process Flow of Calibration Process

5. Examples of Targets in Calibration Process at Shipment

6. Examples of Targets in Calibration Process During Operation

7. Exemplary Computer Configuration

1. Exemplary Configuration of Object Detection System

FIG. 1 is a block diagram illustrating an exemplary configuration of anobject detection system to which the present technology is applied.

An object detection system 1 in FIG. 1 includes a millimeter-wave radar11, a stereo camera 12, and a signal processing apparatus 13, and is asystem that detects an object to become an obstacle individually usingthe millimeter-wave radar 11 and the stereo camera 12. The objectdetection system 1 is mounted in a vehicle such as, for example, anautomobile or a truck.

Note that, in the present embodiment, although a description will begiven of the case where the millimeter-wave radar 11 and the stereocamera 12 are mounted such that the detection direction thereof facesthe front of the vehicle to detect an object in front of the vehicle,the object detection direction is not limited to the front of thevehicle. For example, where the millimeter-wave radar 11 and the stereocamera 12 are mounted so as to face the rear of the vehicle, the objectdetection system 1 detects an object in the rear of the vehicle.

The millimeter-wave radar 11 emits a millimeter wave in a predetermineddirection θ, obtains a reflected wave that collides with a given objectand returns therefrom, and supplies a reflected signal depending on theobtained reflected wave to the signal processing apparatus 13. Themillimeter-wave radar 11 performs millimeter wave scanning within apredetermined angular range in front of the vehicle, and supplies theresultant reflected signal as well as the irradiation direction θ to thesignal processing apparatus 13. In the present embodiment, a unit ofscanning the predetermined angular range one time in the millimeter-waveradar 11 will be referred to as a frame.

The stereo camera 12 includes a right camera 21R and a left camera 21L.The right camera 21R and the left camera 21L are arranged at the sameheight and spaced apart at a predetermined interval in the lateraldirection. The right camera 21R and the left camera 21L capture imagesin a predetermined range in front of the vehicle. The image captured bythe right camera 21R (hereinafter also referred to as a right-cameraimage) and the image captured by the left camera 21L (hereinafter alsoreferred to as a left-camera image) are images with disparity(difference in the lateral direction) due to the difference in thearranged positions. Note that the positional relationship between theright camera 21R and the left camera 21L is accurately calibrated.Hereinafter, the right-camera image and the left-camera image will alsobe referred to as a stereo image when not particularly distinguishedfrom each other.

The signal processing apparatus 13 performs signal processing on asensor signal output from each of sensors of the millimeter-wave radar11 and the stereo camera 12. It is assumed that the millimeter-waveradar 11 and the stereo camera 12 are temporally synchronized to someextent upon sensing.

The signal processing apparatus 13 includes a target detection unit 31,a three-dimensional position calculating unit 32, a target detectionunit 33, a disparity estimation unit 34, a three-dimensional positioncalculating unit 35, a correspondence detection unit 36, a position andattitude estimation unit 37, and a storage unit 38.

For accurate object detection, the object detection system 1 needs toidentify a correspondence relationship between the objects individuallydetected by the millimeter-wave radar 11 and the stereo camera 12. Thatis, the detected objects are represented by the coordinate systems thatdiffer between the millimeter-wave radar 11 and the stereo camera 12. Ina case where the detected objects are identical, the coordinate valuesof the objects detected by the millimeter-wave radar 11 and the stereocamera 12 need to be converted into a predetermined single coordinatesystem so as to be the same values.

The signal processing apparatus 13 performs a process for calculating acorrespondence relationship between the coordinate system of themillimeter-wave radar 11 and the coordinate system of the stereo camera12. In other words, the signal processing apparatus 13 calculates arelationship (position and attitude) of the position of one of themillimeter-wave radar 11 and the stereo camera 12 to the position of theother.

The calibration process for calculating the positional relationshipbetween the millimeter-wave radar 11 and the stereo camera 12, which isperformed by the signal processing apparatus 13, includes a pre-shipmentcalibration process and a calibration process during operation. Thepre-shipment calibration process is executed before the vehicle isshipped. The calibration process during operation is for adjustingdeviation generated after shipment. The deviation after shipment is, forexample, conceivably caused by temporal change, heat, vibration or thelike.

In the pre-shipment calibration process, objects detected by themillimeter-wave radar 11 and the stereo camera 12 are prepared inadvance as targets. The targets in the pre-shipment calibration processare, for example, poles or the like having a texture (pattern) whosepositions can be uniquely identified in a stereo image captured by thestereo camera 12 and reflecting a millimeter wave.

FIG. 2 illustrates an exemplary target used for the pre-shipmentcalibration process.

A target 51 illustrated in FIG. 2 is a cylindrically-shaped pole formedof a material that reflects a millimeter wave. A texture in a checkeredpattern is formed on the cylindrically-shaped outer periphery. In a casewhere the target detection unit 33 calculates the position of the target51 in the stereo image, for example, the target detection unit 33calculates the pixel position of an intersection point 52 of thecheckered pattern by pattern matching, feature extraction or the like.

FIGS. 3 and 4 illustrate an exemplary arrangement of targets 51 in thepre-shipment calibration process.

FIG. 3 is an arrangement diagram of the targets 51 in the pre-shipmentcalibration process when the targets 51 are viewed from above.

In FIG. 3 , the longitudinal direction of the drawing is the Z-axis, thelateral direction of the drawing is the X-axis, and the directionvertical to the drawing is the Y-axis. The longitudinal direction is inthe forward direction (depth direction) of the vehicle. The lateraldirection is in the lateral direction (horizontal direction) of thevehicle.

In the pre-shipment calibration process, the plurality of targets 51 isarranged such that the plurality of targets 51 does not overlap eachother when captured by the stereo camera 12. Desirably, furthermore,each of the plurality of targets 51 is arranged so as not to be at thesame position as the other targets 51 in either one of the X-axisdirection and the Z-axis direction, as illustrated in FIG. 3 .

FIG. 4 is an arrangement diagram of the targets 51 in the pre-shipmentcalibration process when the targets 51 are viewed from the lateraldirection.

In FIG. 4 , the lateral direction of the drawing is the Z-axis, thelongitudinal direction of the drawing is the Y-axis, and the directionvertical to the drawing is the X-axis.

Assume that the millimeter-wave radar 11 is arranged so as to irradiatethe XZ plane at a height h from the ground with the millimeter wave. Inthis case, the plurality of targets 51 is arranged such thatintersection points 52 of the targets 51 are at the height h of themillimeter wave, as illustrated in FIG. 4 . In other words, theintersection points 52 in the checkered pattern of the targets 51 areformed so as to match the height h of the millimeter wave emitted by themillimeter-wave radar 11.

Note that although the stereo camera 12 may also be arranged such thatthe center of the imaging is at the position of the height h from theground, which is the same as the intersection points 52 in the checkeredpattern, so as to match the height position of the millimeter-wave radar11, the stereo camera 12 is not necessarily arranged to match the heightposition of the millimeter-wave radar 11.

In the calibration process during operation, on the other hand, it isnot possible to specify a predetermined fixed object as the target.Accordingly, an object that exists on a path in which the vehicletravels serves as the target. For example, a pedestrian or a pole suchas a sign or a utility pole serves as the target in the calibrationprocess during operation.

Note that calculation of the positional relationship between themillimeter-wave radar 11 and the stereo camera 12 needs positioninformation of a plurality of targets at different positions. Theposition information of the plurality of targets may be individuallyobtained by the millimeter-wave radar 11 and the stereo camera 12detecting the plurality of targets in one frame or by preparing aplurality of frames, each of which captures a single target.

The target detection unit 31 and the three-dimensional positioncalculating unit 32 on the millimeter-wave radar 11 side will bedescribed with reference to FIG. 5 .

The target detection unit 31 detects the positions of the targets infront of the vehicle on the basis of the reflected signals and theirradiation directions θ supplied from the millimeter-wave radar 11.More specifically, the target detection unit 31 detects, as the targetdetection positions, peak positions whose reflected signal intensitiesare equal to or higher than a predetermined intensity on the basis of areflection intensity map in which the intensities of the reflectedsignals and the irradiation directions θ are associated with each other.The target detection positions are represented by a polar coordinatesystem including a distance L based on the intensity of the reflectedsignal and the irradiation direction θ. The detected target detectionpositions are supplied to the three-dimensional position calculatingunit 32.

In FIG. 5 , a black triangle expanding from the millimeter-wave radar 11indicates an irradiation range of the millimeter wave, while thepositions where the targets are detected are indicated in white. Thehigher the intensity of the reflected signal, the more it is expressedin white.

The three-dimensional position calculating unit 32 converts the targetdetection positions represented by the polar coordinate system andsupplied from the target detection unit 31 into the target detectionpositions on a three-dimensional coordinate system in which the forwarddirection (depth direction) of the vehicle is the Z-axis, the lateraldirection (horizontal direction) is the X-axis, and the longitudinaldirection (vertical direction) is the Y-axis.

That is, the target detection positions represented by the polarcoordinate system including the distance L based on the intensity of thereflected signal and the irradiation direction θ are converted into anorthogonal coordinate system by the three-dimensional positioncalculating unit 32 so as to be converted into the target detectionpositions on the XZ plane of the three-dimensional coordinate system.

Here, the calculated target detection positions are the positions on thethree-dimensional coordinate system relative to the millimeter-waveradar 11, and the three-dimensional coordinate system relative to themillimeter-wave radar 11 will also be referred to as a radarthree-dimensional coordinate system to be distinguished from athree-dimensional coordinate system relative to the stereo camera 12 tobe described later.

The three-dimensional position calculating unit 32 supplies thecalculated target detection positions represented by the radarthree-dimensional coordinate system to the correspondence detection unit36.

The target detection unit 33, the disparity estimation unit 34, and thethree-dimensional position calculating unit 35 on the stereo camera 12side will be described with reference to FIG. 6 .

The target detection unit 33 detects the positions of the targets on thetwo-dimensional coordinate system including the X-axis and the Y-axis byperforming, on the stereo image supplied from the right camera 21R andthe left camera 21L, pattern matching (image recognition process) with apreliminarily registered pattern (shape or texture) or a featuredetection process that detects the features of the target image.

Relative to either one of the right-camera image supplied from the rightcamera 21R and the left-camera image supplied from the left camera 21L(left-camera image in the present embodiment), the target detection unit33 detects the positions of the intersection points 52 of the targets 51in the left-camera image with pixel-level precision, and supplies thepositions to the three-dimensional position calculating unit 35.

The disparity estimation unit 34 calculates disparity from theright-camera image supplied from the right camera 21R and theleft-camera image supplied from the left camera 21L and supplies thecalculation result to the three-dimensional position calculating unit 35as disparity information.

FIG. 6 illustrates a disparity image expressed by luminance values. Theluminance values are higher as the disparity relative to the left-cameraimage, which is calculated from the right-camera image and theleft-camera image, is greater. The disparity image in FIG. 6 indicatesthat the higher the luminance value, the closer the distance to thecorresponding target 51.

The three-dimensional position calculating unit 35 calculates thepositions (distances) in the Z-axis direction, which is the forwarddirection of the vehicle, on the basis of the disparity information ofthe targets supplied from the disparity estimation unit 34. Then, thethree-dimensional position calculating unit 35 calculates the targetdetection positions on the three-dimensional coordinate system from thepositions of the calculated targets in the Z-axis direction and thepositions of the targets on the two-dimensional coordinate system (XYplane) supplied from the target detection unit 33. In thethree-dimensional coordinate system, the forward direction (depthdirection) of the vehicle is the Z-axis, the lateral direction(horizontal direction) is the X-axis, and the longitudinal direction(vertical direction) is the Y-axis. The target detection positionscalculated here are the positions on the three-dimensional coordinatesystem relative to the stereo camera 12, and have the same the axisdirections as the radar three-dimensional coordinate system but have adifferent point of origin from the radar three-dimensional coordinatesystem. The three-dimensional coordinate system relative to the stereocamera 12 will also be referred to as a camera three-dimensionalcoordinate system to be distinguished from the radar three-dimensionalcoordinate system described above. Further, where distinction betweenthe radar three-dimensional coordinate system and the camerathree-dimensional coordinate system is not particularly necessary, bothwill also be collectively referred to as a sensor coordinate system.

The three-dimensional position calculating unit 35 supplies thecalculated target detection positions represented by the camerathree-dimensional coordinate system to the correspondence detection unit36.

The correspondence detection unit 36 detects correspondencerelationships between the targets detected on the radarthree-dimensional coordinate system and the targets detected on thecamera three-dimensional coordinate system. In other words, thecorrespondence detection unit 36 detects which target detected on thecamera three-dimensional coordinate system corresponds to the targetdetected on the radar three-dimensional coordinate system.

In the pre-shipment calibration process using the targets preliminarilyprepared, the arrangement of the targets is known in advance. In thiscase, the correspondence detection unit 36 obtains prior arrangementinformation of the targets from the storage unit 38 and individuallycollates the target detection positions detected on the radarthree-dimensional coordinate system and the target detection positionsdetected on the camera three-dimensional coordinate system with theobtained target prior arrangement information. After identifying thetargets, the correspondence detection unit 36 detects the correspondencerelationships between the targets detected on the radarthree-dimensional coordinate system and the targets detected on thecamera three-dimensional coordinate system.

Specifically, as illustrated in FIG. 7 , for example, the correspondencedetection unit 36 detects that a target detection position a detected onthe radar three-dimensional coordinate system corresponds to a targetposition 1 in the target prior arrangement information, a targetdetection position b corresponds to a target position 2, and similarly,subsequent target detection positions c to g correspond one-to-one totarget positions 3 to 7.

Further, the correspondence detection unit 36 detects that a targetdetection position A detected on the camera three-dimensional coordinatesystem corresponds to the target position 1 in the target priorarrangement information, a target detection position B corresponds tothe target position 2, and similarly, subsequent target detectionpositions C to G correspond one-to-one to the target positions 3 to 7.

As a result, the correspondence detection unit 36 detects that thetarget at the target detection position a detected on the radarthree-dimensional coordinate system and the target at the targetdetection position A detected on the camera three-dimensional coordinatesystem correspond to each other. Similarly, the correspondence detectionunit 36 individually detects that the targets at the target detectionpositions b to g detected on the radar three-dimensional coordinatesystem and the targets at the target detection positions B to G detectedon the camera three-dimensional coordinate system correspond to eachother.

By contrast, in the calibration process during operation where it is notpossible to obtain the target arrangement information, thecorrespondence detection unit 36 detects a correspondence relationshipbetween a target detected on the radar three-dimensional coordinatesystem and a target detected on the camera three-dimensional coordinatesystem by comparing the target detection position detected on the radarthree-dimensional coordinate system with the target detection positiondetected on the camera three-dimensional coordinate system on the basisof the positional relationship which has already been obtained throughthe pre-shipment calibration process or calibration process duringoperation executed before.

The position and attitude estimation unit 37 calculates a positionalrelationship between the millimeter-wave radar 11 and the stereo camera12 using the plurality of targets whose correspondence relationshipshave been identified by the correspondence detection unit 36.

Specifically, as illustrated in FIG. 8 , the position of the k-th target(0<k<K+1) among K targets whose correspondence relationships have beenidentified by the correspondence detection unit 36 is represented asP_(MMW)(k)=[X_(MMW)(k) Y_(A) Z_(MMW)(k)]^(T) on the radarthree-dimensional coordinate system and represented asP_(cam)(k)=[X_(cam)(k) Y_(cam) (k) Z_(cam)(k)]^(T) on the camerathree-dimensional coordinate system, where T represents transpositionand Y_(A) represents a predetermined fixed value.

The position and attitude estimation unit 37 calculates a rotationmatrix R and a translation vector V of the equation (1) by substitutingeach of the K targets into the equation (1) representing the positionalrelationship between the target position P_(MMW)(k) on the radarthree-dimensional coordinate system and the target position P_(cam)(k)on the camera three-dimensional coordinate system, and solving anoptimization problem using the method of least squares and the like.P _(cam)(k)=R·P _(MMW)(k)+V  (1)

In the equation (1), k is a variable (0<k<K+1) that identifies apredetermined one of the plurality (K) of targets, P_(cam)(k) representsthe target detection position of the k-th target detected on the camerathree-dimensional coordinate system, and P_(MMW)(k) represents thetarget detection position of the k-th target detected on the radarthree-dimensional coordinate system.

The equation (1) corresponds to the equation for converting the targetdetection position P_(MMW)(k) of the k-th target detected on the radarthree-dimensional coordinate system into the target detection positionP_(cam)(k) on the camera three-dimensional coordinate system. Therotation matrix R represents the attitude of the millimeter-wave radar11 relative to the stereo camera 12. The translation vector V representsthe position of the millimeter-wave radar 11 relative to the stereocamera 12.

The number of variables of the rotation matrix R is three and the numberof variables of the translation vector V is three. Therefore, as long asat least six target detection positions can be obtained, the rotationmatrix R and translation vector V of the equation (1) can be calculated.Note that the rotation matrix R can not only be solved by using themethod of least squares but also by being represented with quaternions.

The storage unit 38 stores the positional relationship information(calibration information) on the millimeter-wave radar 11 and the stereocamera 12 calculated by the position and attitude estimation unit 37.Specifically, the storage unit 38 stores the rotation matrix R and thetranslation vector V of the equation (1) supplied from the position andattitude estimation unit 37.

The object detection system 1 is configured as above.

2. Detailed Description of Correspondence Detection Process

<First Correspondence Detection Process>

Next, a first correspondence detection process using the priorarrangement information of the targets that the correspondence detectionunit 36 performs will be described in more detail.

As illustrated in FIG. 9 , it is assumed that the position of the k-thtarget is represented as P_(MAP)(k)=[X_(MAP)(k) Y_(MAP)(k)Z_(MAP)(k)]^(T) in the prior arrangement information of the targets on aworld coordinate system with a predetermined location as the point oforigin, while the position of the k-th target is represented asP_(Det)(k)=[X_(Det)(k) Y_(Det)(k) Z_(Det)(k)]^(T) on the sensorcoordinate system of the millimeter-wave radar 11 or the stereo camera12.

Note that where the sensor coordinate system is the radarthree-dimensional coordinate system, Y_(Det)(k) is a fixed value asdescribed above. Further, although the number of targets is K, there arecases where K or more targets are detected due to the influence ofdisturbance or the like on the sensor coordinate system of themillimeter-wave radar 11 or the stereo camera 12. In the example in FIG.9 , although there are five targets in the prior arrangementinformation, a target position f is detected as a target on the sensorcoordinate system due to noise, for example, resulting in the detectionof six targets at the target detection positions a to f.

In this way, the detection of the correspondence relationships betweenthe five target positions 1 to 5 on the world coordinate system and thesix target detection positions a to f on the sensor coordinate systemcan be solved by being regarded as a graph matching problem that findscorrespondence relationships where three-dimensional points on differentcoordinate systems overlap the most.

The possible corresponding combinations of the five target positions 1to 5 on the world coordinate system and the six target detectionpositions a to f on the sensor coordinate system are as illustrated inFIG. 10 .

When the correspondence relationships (connections) between the fivetarget positions 1 to 5 on the world coordinate system and the sixtarget detection positions a to f on the sensor coordinate system arerepresented by a matrix variable X with M rows and N columns, thecorrespondence relationships can be represented by the followingequation (2).

[Math.  1] $\begin{matrix}{{X = \begin{bmatrix}x_{0,0} & \cdots & x_{0,N} \\\vdots & \ddots & \vdots \\x_{M,0} & \cdots & x_{M,N}\end{bmatrix}}{{x_{i,j} \in \left\{ {0,1} \right\}},{{\Sigma_{i}^{M}x_{i,j}} = 1}}} & (2)\end{matrix}$

In the equation (2), M is the number of targets on the world coordinatesystem (M=5 in the example in FIG. 9 ), while N is the number of targetson the sensor coordinate system (N=6 in the example in FIG. 9 ).Further, the index i of x represents a variable (0<i<M+1) thatidentifies a target on the world coordinate system, while the index j ofx represents a variable (0<j<N+1) that identifies a target on the sensorcoordinate system. x_(i,j) represents whether the i-th target on theworld coordinate system and the j-th target on the sensor coordinatesystem are connected, and is a variable that takes “1” when connectedwhile “0” when not connected.

For example, as indicated by the bold solid lines in FIG. 11 , where thetarget position 1 on the world coordinate system and the targetdetection position a on the sensor coordinate system, the targetposition 2 on the world coordinate system and the target detectionposition b on the sensor coordinate system, the target position 3 on theworld coordinate system and the target detection position c on thesensor coordinate system, the target position 4 on the world coordinatesystem and the target detection position d on the sensor coordinatesystem, and the target position 5 on the world coordinate system and thetarget detection position e on the sensor coordinate system correspondto each other, the matrix variable X representing the correspondencerelationships is represented as below.

[Math.  2] $X = \begin{bmatrix}1 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 0\end{bmatrix}$

Then, the correspondence detection unit 36 obtains X that maximizes ascore function score (X) using the matrix variable X represented by theequation (2). The score function score (X) is represented by thefollowing equation (3).

[Math.  3] $\begin{matrix}{{{\max\mspace{14mu}{{score}(X)}} = {\sum\limits_{{i\; 1},{i\; 2},{j\; 1},{j\; 2}}{{S\left( {l_{{i\; 1},{i\; 2}},h_{{j\; 1},{j\; 2}}} \right)}x_{{i\; 1},{j\; 1}}}}},x_{{i\; 2},{j\; 2}}} & (3)\end{matrix}$

In the equation (3), i1 and i2 are variables that identify the targetson the world coordinate system, while j1 and j2 are variables thatidentify the targets on the sensor coordinate system. l_(i1,i2)represents the length of a line segment connecting P_(MAP)(i1) andP_(MAP)(i2) on the world coordinate system, while h_(j1,j2) representsthe length of a line segment connecting P_(Det)(j1) and P_(Det)(j2) onthe sensor coordinate system.

S(l_(i1,i2),h_(j1,j2)) represents the similarity between the linesegment length l_(i1,i2) and the line segment length h_(j1,j2), andbecomes a greater value as the line segment length l_(i1,i2) and theline segment length h_(j1,j2) are closer values. For the similarityS(l_(i1,i2),h_(j1,j2)), the following equation (4) that uses thedifference d(l_(i1,i2),h_(j1,j2)) between the line segment lengthl_(i1,i2) and the line segment length h_(j1,j2) can be employed, forexample.[Math. 4]S(l _(i1,i2) ,h _(j1,j2))=2^(−|d(l) ^(i1,i2) ^(,h) ^(j1,j2) ^()|)  (4)

The score function score (X) calculated as described above is based onthe idea that, for example, when the line segment between the targets 1and 4 on the world coordinate system and the line segment between thetargets a and d on the sensor coordinate system correspond to eachother, the lengths of both line segments, that is, l_(1,4) and l_(a,d)are approximately equal to each other and the differenced(l_(i1,i2),h_(j1,j2))==d(l_(1,4),h_(a,d)) becomes small, as illustratedin FIG. 12 .

<Second Correspondence Detection Process>

The above-described first correspondence detection process is adetection method using the prior arrangement information of the targets,but it is also possible to detect the correspondence relationshipsbetween the targets detected on the radar three-dimensional coordinatesystem and the targets detected on the camera three-dimensionalcoordinate system without using the prior arrangement information of thetargets.

For example, as illustrated in FIG. 13 , the correspondence detectionunit 36 slides at least one of the target position P_(MMW)(k) on theradar three-dimensional coordinate system and the target positionP_(cam)(k) on the camera three-dimensional coordinate system by apredetermined amount for superimposition, so that the targets arrangedclosest to each other can correspond to each other.

3. Calibration Process During Operation

Next, the calibration process during operation will be described.

FIG. 14 is a block diagram illustrating a detailed exemplaryconfiguration of the target detection unit 31 on the millimeter-waveradar 11 side where the calibration process during operation isexecuted.

The target detection unit 31 includes a motion detection unit 71, a peakdetection unit 72, and an AND operation unit 73.

The motion detection unit 71 includes a storage unit that stores areflected signal of at least one previous frame. The motion detectionunit 71 detects the motion of a peak position by comparing reflectedsignals of the current frame supplied from the millimeter-wave radar 11with reflected signals of the previous frame inputted immediatelybefore. The motion detection unit 71 supplies the peak position whosemotion is detected to the AND operation unit 73.

The peak detection unit 72 detects a peak position whose reflectedsignal intensity is equal to or higher than a predetermined intensityamong the reflected signals of the current frame supplied from themillimeter-wave radar 11 and supplies the detection result to the ANDoperation unit 73.

The AND operation unit 73 performs an AND operation on the peak positionsupplied from the motion detection unit 71 and the peak positionsupplied from the peak detection unit 72. In other words, of the peakposition supplied from the peak detection unit 72, the AND operationunit 73 extracts the peak position supplied from the motion detectionunit 71, that is, only the peak position whose motion is detected, andsupplies the extraction result to the three-dimensional positioncalculating unit 32 as the target detection position.

FIG. 15 is a block diagram illustrating a detailed exemplaryconfiguration of the target detection unit 33 on the stereo camera 12side where the calibration process during operation is executed.

The target detection unit 33 includes a motion area detection unit 81,an image recognition unit 82, an AND operation unit 83, and a centerposition calculation unit 84.

The motion area detection unit 81 includes a storage unit that stores astereo image of at least one previous frame. The motion area detectionunit 81 detects a motion area in the stereo image by comparing thestereo image of the current frame supplied from the stereo camera 12with the stereo image of the previous frame inputted immediately before.The motion area in the stereo image can be detected by using motionvector estimation, frame difference or the like. The motion areadetection unit 81 supplies the detected motion area to the AND operationunit 83.

The image recognition unit 82 detects a target area by performing imagerecognition on the stereo image of the current frame supplied from thestereo camera 12. For example, in the case of detecting a pedestrian(human) as a target, the target area can be detected by performing theimage recognition process for recognizing the shape of the human(silhouette) or the face thereof. The image recognition unit 82 suppliesthe detected target area to the AND operation unit 83.

The AND operation unit 83 performs an AND operation on the motion areasupplied from the motion area detection unit 81 and the target areasupplied from the image recognition unit 82. In other words, of themotion area supplied from the motion area detection unit 81, the ANDoperation unit 83 extracts the target area supplied from the imagerecognition unit 82, that is, only the target area whose motion isdetected, and supplies the extraction result to the center positioncalculation unit 84.

The center position calculation unit 84 calculates a pixel positionserving as the center of the target area supplied from the AND operationunit 83, and supplies the calculated pixel position to thethree-dimensional position calculating unit 32 as the target detectionposition.

<Specific Example of Calibration Process During Operation>

The calibration process during operation will be specifically describedwith reference to FIGS. 16 to 21 .

A description will be given of an example where while, for example, avehicle mounting the object detection system 1 is stopped, the objectdetection system 1 detects a pedestrian 101 in front illustrated in FIG.16 as a target and executes the calibration process during operation.

The detection range of the millimeter-wave radar 11 and the stereocamera 12 includes the pedestrian 101 and two fixed objects 102-1 and102-2. The pedestrian 101 is in the middle of moving in the rightdirection in the figure, and the fixed objects 102-1 and 102-2 areobjects that do not move.

In the target detection unit 31 on the millimeter-wave radar 11 side,the peak detection unit 72 detects peak positions 111 to 113 fromreflected signals of the current frame supplied from the millimeter-waveradar 11 as illustrated in FIG. 17 , and supplies the detection resultto the AND operation unit 73. The peak position 111 corresponds to thepedestrian 101 in FIG. 16 , and the peak positions 112 and 113correspond to the fixed objects 102-1 and 102-2, respectively.

Meanwhile, the motion detection unit 71 compares the reflected signalsof the current frame with reflected signals of the previous frameinputted immediately before, and supplies only the peak position 111 tothe AND operation unit 73 as the peak position whose motion is detected.

Of the peak positions 111 to 113 supplied from the peak detection unit72, the AND operation unit 73 supplies only the peak position 111supplied from the motion detection unit 71 to the three-dimensionalposition calculating unit 32 as the target detection position.

Meanwhile, in the target detection unit 33 on the stereo camera 12 side,the image recognition unit 82 detects a target area 121 by performingthe image recognition process for recognizing the human shape and faceon a stereo image of the current frame as illustrated in FIG. 18 . Thetarget area 121 corresponds to the pedestrian 101 in FIG. 16 .

The motion area detection unit 81 detects a motion area 122 in thestereo image by comparing the stereo image of the current frame suppliedfrom the stereo camera 12 with a stereo image of the previous frameinputted immediately before. The motion area 122 detected here alsocorresponds to the pedestrian 101 in FIG. 16 .

Note that the stereo image on which the target detection unit 33performs the image recognition process and the stereo image from whichthe motion area detection unit 81 detects the motion area use the sameleft-camera image as the disparity image.

The AND operation unit 83 performs an AND operation on the motion area122 supplied from the motion area detection unit 81 and the target area121 supplied from the image recognition unit 82, and supplies theresultant target area 121 to the center position calculation unit 84.

The center position calculation unit 84 calculates a center pixelposition 123 of the target area 121 supplied from the AND operation unit83, and supplies the calculated center pixel position 123 to thethree-dimensional position calculating unit 32 as the target detectionposition.

Next, as illustrated in FIG. 19 , the three-dimensional positioncalculating unit 35 on the stereo camera 12 side calculates a targetdetection position 131 on the camera three-dimensional coordinate systemfrom the disparity information relative to the left-camera imagesupplied from the disparity estimation unit 34 and the target detectionposition 123 supplied from the center position calculation unit 84 ofthe target detection unit 33, and supplies the target detection position131 to the correspondence detection unit 36. In the camerathree-dimensional coordinate system, the forward direction of thevehicle is the Z-axis, the lateral direction is the X-axis, and thelongitudinal direction is the Y-axis.

Meanwhile, the three-dimensional position calculating unit 32 on themillimeter-wave radar 11 side converts the target detection position 111represented by the polar coordinate system and supplied from the targetdetection unit 31 into a target detection position 132 on the radarthree-dimensional coordinate system, and supplies the target detectionposition 132 to the correspondence detection unit 36. In the radarthree-dimensional coordinate system, the forward direction of thevehicle is the Z-axis, the lateral direction is the X-axis, and thelongitudinal direction is the Y-axis.

The correspondence detection unit 36 obtains positional relationshipinformation of the millimeter-wave radar 11 and the stereo camera 12,specifically, the rotation matrix R and the translation vector V of theequation (1) that have been calculated by the pre-shipment calibrationprocess and stored in the storage unit 38.

Then, the correspondence detection unit 36 calculates a target detectionposition 133 corrected from the target detection position 132 on theradar three-dimensional coordinate system supplied from thethree-dimensional position calculating unit 32 into the position on thecamera three-dimensional coordinate system by using the positionalrelationship information of the millimeter-wave radar 11 and the stereocamera 12 at the present time, as illustrated in FIG. 19 .

Subsequently, the correspondence detection unit 36 detects acorrespondence relationship between the target detected on the stereocamera 12 side and the target detected on the millimeter-wave radar 11side by comparing the target detection position 131 on the camerathree-dimensional coordinate system supplied from the three-dimensionalposition calculating unit 35 on the stereo camera 12 side with thetarget detection position 133 on the millimeter-wave radar 11 sidecorrected into the camera three-dimensional coordinate system.

The correspondence detection unit 36 recognizes that the targets whosecoordinate positions are closest to each other are the targets thatcorrespond to each other. Although the number of detected targets is onein the example in FIG. 19 , even when a plurality of targets isdetected, it is possible to detect correspondence relationships easilybecause the position-corrected detection positions are compared usingthe positional relationship information calculated by the pre-shipmentcalibration process.

When the process described with reference to FIGS. 17 to 19 is theprocess at time t, for example, the signal processing apparatus 13executes the process with a plurality of frames (N frames), asillustrated in FIG. 20 . With this configuration, where a correspondingpoint detected in each frame is one point, the corresponding points of Npoints at different detection positions are detected in the N frames andaccumulated in the storage unit 38.

As described above, at least six target detection positions arenecessary to calculate the rotation matrix R and the translation vectorV of the equation (1). These six target detection positions may beobtained from a total of six frames each including one point or may beobtained from a total of three frames each including two points, forexample. Note that a greater number of corresponding points for solvingthe equation (1) is desirable to improve the calibration precision.Therefore, the number of frames N is 6 or greater and desirably of agreater value.

In the calibration process during operation, furthermore, time t to t+Nof the N frames for solving the equation (1), which are illustrated inFIG. 20 , are not necessarily temporally continuous. For example, it isalso possible to execute the calibration process during operationdescribed above using corresponding points detected in 10 frames in oneday that satisfies a predetermined condition and corresponding pointsdetected in 20 frames in another day that satisfies the predeterminedcondition.

Further, the signal processing apparatus 13 can select a frame to beused for the calibration process during operation to improve thecalibration precision.

Specifically, the signal processing apparatus 13 selects a frame inwhich another target detection position does not exist within apredetermined range (distance) of a detected target detection positionas a frame to be used for the calibration process during operation andaccumulates the frame in the storage unit 38.

For example, target detection positions 141 and 142 are detected in aframe A in FIG. 21 . Outside a predetermined range 143 of the targetdetection position 141, another target detection position 142 exists. Insuch a case, the frame A in FIG. 21 is selected as a frame to be usedfor the calibration process during operation.

In a frame B in FIG. 21 , by contrast, another target detection position142 exists within the predetermined range 143 of the target detectionposition 141. In this case, the frame B in FIG. 21 is excluded from aframe to be used for the calibration process during operation.

Further, where a predetermined number or more of targets are detected inone frame as in a frame C in FIG. 21 , the signal processing apparatus13 also excludes the frame from the frame to be used for the calibrationprocess during operation.

In this way, unlike the pre-shipment calibration process, since a targetprepared in advance is not used in the calibration process duringoperation, a frame (target) in which a corresponding point is detectedand with which higher precision can be obtained is selected, and thepositional relationship information of the millimeter-wave radar 11 andthe stereo camera 12 is recalculated. Note that the frame selection maybe performed by the target detection unit 31 or 33, or may be performedby the correspondence detection unit 36.

4. Process Flow of Calibration Process

Next, the calibration process executed by the signal processingapparatus 13 will be described with reference to the flowchart in FIG.22 .

First, the signal processing apparatus 13 executes the calibrationprocess at shipment in step S1. Although the details of this processwill be described later with reference to the flowchart in FIG. 23 , therotation matrix R and the translation vector V of the equation (1),which serve as the positional relationship information of themillimeter-wave radar 11 and the stereo camera 12, are calculated andstored in the storage unit 38 through this process.

The calibration process at shipment in step S1 is executed where a user(operator) issues instructions to start the calibration through anoperation panel or the like in a factory that manufactures the vehiclemounting the object detection system 1, or a sales outlet such as adealer, for example. Alternatively, the calibration process at shipmentmay also be automatically executed where (it is detected that) thevehicle is stopped at a place which is an environment adequate for thecalibration. After the calibration process at shipment ends, the vehicleis shipped and delivered to an owner (driver).

In step S2, the signal processing apparatus 13 determines whether tostart the calibration process during operation. For example, the signalprocessing apparatus 13 determines the start of the calibration processduring operation when a predetermined start condition is satisfied, forexample, when a certain period or longer has elapsed since the previouscalibration process at shipment or calibration process during operation,the predetermined number or more of corresponding points are accumulatedin the storage unit 38 as described in FIG. 20 , or the amount ofdeviation of the position-corrected corresponding points of themillimeter-wave radar 11 and the stereo camera 12 becomes equal to orgreater than a predetermined value at all times (equal to or greaterthan a predetermined number of times).

Further, the signal processing apparatus 13 determines not to start thecalibration process during operation where the vehicle is not horizontalto the road surface (tilted), the vehicle is moving at high speed, orthe number of targets detected at a time is equal to or greater than apredetermined value, or in case of the environment condition such as inbad weather, retrograde, darkness or the like where reliability of thestereo image captured by the stereo camera 12 is low, or the environmentcondition where the reliability of the millimeter-wave radar 11 is lowsince the vehicle is in a place (for example, a tunnel) where multiplereflection of the millimeter wave of the millimeter-wave radar 11 islikely to occur. Whether or not the vehicle is in a place where multiplereflection is likely to occur can be determined on the basis of, forexample, a received GPS signal.

In step S2, in a case where it is determined that the calibrationprocess during operation does not start, the process returns to step S2and the process in step S2 is repeated until it is determined that thecalibration process during operation starts.

On the other hand, in a case where it is determined in step S2 that thecalibration process during operation stats, the process proceeds to stepS3 and the signal processing apparatus 13 executes the calibrationprocess during operation. Although the details of this process will bedescribed later with reference to the flowchart in FIG. 24 , therotation matrix R and the translation vector V of the equation (1),which serve as the positional relationship information of themillimeter-wave radar 11 and the stereo camera 12, are recalculated andoverwritten (updated) in the storage unit 38 through this process.

FIG. 23 is a flowchart for describing the detailed calibration processat shipment in step S1 described above.

In step S21, the target detection unit 31 on the millimeter-wave radar11 side detects the positions of the targets in front of the vehicle onthe basis of the reflected signals and the irradiation directions θsupplied from the millimeter-wave radar 11. The target detectionpositions detected by the target detection unit 31 are each representedby the polar coordinate system including the distance L based on theintensity of the reflected signal and the irradiation direction θ andsupplied to the three-dimensional position calculating unit 32.

In step S22, the three-dimensional position calculating unit 32 on themillimeter-wave radar 11 side converts the target detection positionsrepresented by the polar coordinate system and supplied from the targetdetection unit 31 into the target detection positions on the radarthree-dimensional coordinate system.

In step S23, the target detection unit 33 on the stereo camera 12 sidedetects the positions of the targets on the two-dimensional coordinatesystem by performing image processing such as the pattern matching orfeature detection process on the stereo image.

In step S24, the disparity estimation unit 34 calculates disparity fromthe right-camera image supplied from the right camera 21R and theleft-camera image supplied from the left camera 21L and supplies thedisparity to the three-dimensional position calculating unit 35 asdisparity information.

In step S25, the three-dimensional position calculating unit 35calculates the target detection positions on the camerathree-dimensional coordinate system from the disparity informationsupplied from the disparity estimation unit 34 and the target detectionpositions on the two-dimensional coordinate system supplied from thetarget detection unit 33.

Note that the processes in steps S21 to S25 can be executed sequentiallyas described above, or alternatively, the processes in steps S21 and S22can be executed in parallel with the processes in steps S23 to S25.

In step S26, the correspondence detection unit 36 detects correspondencerelationships between the targets detected on the radarthree-dimensional coordinate system and the targets detected on thecamera three-dimensional coordinate system by executing theabove-described first correspondence detection process. Specifically,the correspondence detection unit 36 identifies the targets by collatingthe target detection positions detected on the radar three-dimensionalcoordinate system with the target prior arrangement information.Further, the correspondence detection unit 36 identifies the targets bycollating the target detection positions detected on the camerathree-dimensional coordinate system with the target prior arrangementinformation. Then, the correspondence detection unit 36 detects whichtarget detected on the camera three-dimensional coordinate systemcorresponds to the target detected on the radar three-dimensionalcoordinate system on the basis of the result of collation with thetarget prior arrangement information.

Note that in step S26, the above-described second correspondencedetection process may be executed instead of the first correspondencedetection process.

In step S27, the position and attitude estimation unit 37 calculates apositional relationship between the millimeter-wave radar 11 and thestereo camera 12 by substituting the target detection positions of theplurality of targets whose correspondence relationships are identifiedby the correspondence detection unit 36 into the equation (1) andsolving the equation (1) using the method of least squares and the like.In this way, the rotation matrix R and the translation vector V of theequation (1) are calculated.

In step S28, the position and attitude estimation unit 37 stores thecalculated rotation matrix R and translation vector V in the storageunit 38.

Now, the calibration process at shipment in step S1 ends.

Next, the details of the calibration process during operation in step S3described above will be described with reference to the flowchart inFIG. 24 .

In step S41, the target detection unit 31 on the millimeter-wave radar11 side detects the position of the target whose motion is detected onthe basis of the reflected signal and the irradiation direction θsupplied from the millimeter-wave radar 11.

More specifically, the motion detection unit 71 detects the motion of apeak position by comparing reflected signals of the current frame withreflected signals of the previous frame inputted immediately before.Further, the peak detection unit 72 detects a peak position from thereflected signals of the current frame and supplies the peak position tothe AND operation unit 73. Then, of the peak position supplied from thepeak detection unit 72, the AND operation unit 73 extracts only the peakposition supplied from the motion detection unit 71 and supplies theextraction result to the three-dimensional position calculating unit 32as the target detection position whose motion is detected.

In step S42, the three-dimensional position calculating unit 32 on themillimeter-wave radar 11 side converts the target detection positionrepresented by the polar coordinate system and supplied from the targetdetection unit 31 into the target detection position on the radarthree-dimensional coordinate system.

In step S43, the target detection unit 33 on the stereo camera 12 sidedetects the position of the target whose motion is detected byperforming image processing such as the pattern matching or featuredetection process on the stereo image.

More specifically, the motion area detection unit 81 detects a motionarea in the stereo image by comparing the current stereo image with thestereo image of the previous frame inputted immediately before. Theimage recognition unit 82 detects a target area by performing imagerecognition on the current stereo image supplied from the stereo camera12. Of the motion area supplied from the motion area detection unit 81,the AND operation unit 83 extracts the target area supplied from theimage recognition unit 82 and supplies the extraction result to thecenter position calculation unit 84. The center position calculationunit 84 calculates a center pixel position of the target area suppliedfrom the AND operation unit 83, and supplies the calculated center pixelposition to the three-dimensional position calculating unit 32 as thetarget detection position whose motion is detected.

In step S44, the disparity estimation unit 34 calculates disparity fromthe right-camera image supplied from the right camera 21R and theleft-camera image supplied from the left camera 21L and supplies thedisparity to the three-dimensional position calculating unit 35 asdisparity information.

In step S45, the three-dimensional position calculating unit 35 on thestereo camera 12 side calculates the target detection position on thecamera three-dimensional coordinate system from the disparityinformation supplied from the disparity estimation unit 34 and thetarget detection position on the two-dimensional coordinate systemsupplied from the target detection unit 33.

In step S46, the correspondence detection unit 36 obtains the positionalrelationship information of the millimeter-wave radar 11 and the stereocamera 12 stored in the storage unit 38, specifically, the rotationmatrix R and the translation vector V of the equation (1). Here, in thecalibration process during operation performed for the first time, thepositional relationship information obtained from the storage unit 38 isthe data calculated in the pre-shipment calibration process. In thesecond and following calibration processes during operation, thepositional relationship information is the data updated in the previouscalibration process during operation.

In step S47, the correspondence detection unit 36 calculates the targetdetection position corrected from the target detection position 132 onthe radar three-dimensional coordinate system supplied from thethree-dimensional position calculating unit 32 into the position on thecamera three-dimensional coordinate system using the obtained positionalrelationship information.

In step S48, the correspondence detection unit 36 detects acorrespondence relationship between the target detected on the stereocamera 12 side and the target detected on the millimeter-wave radar 11side by comparing the target detection position on the camerathree-dimensional coordinate system supplied from the three-dimensionalposition calculating unit 35 on the stereo camera 12 side with thetarget detection position on the millimeter-wave radar 11 side correctedinto the camera three-dimensional coordinate system.

In step S49, the position and attitude estimation unit 37 calculates apositional relationship between the millimeter-wave radar 11 and thestereo camera 12 by substituting a plurality of target detectionpositions of the target whose correspondence relationships areidentified through the process in step S48 into the equation (1) andsolving the equation (1).

In step S50, the position and attitude estimation unit 37 determineswhether the new positional relationship information is within apredetermined range of the current positional relationship informationby comparing the current positional relationship information stored inthe storage unit 38 with the positional relationship information newlycalculated in step S49.

In step S50, where it is determined that the new positional relationshipinformation is within the predetermined range of the current positionalrelationship information, the process proceeds to step S51. The positionand attitude estimation unit 37 overwrites the current positionalrelationship information stored in the storage unit 38 with the newpositional relationship information for storage, and the calibrationprocess during operation ends.

On the other hand, where it is determined in step S50 that the newpositional relationship information is not within the predeterminedrange of the current positional relationship information, the process instep S51 is skipped and the calibration process during operation ends.

In the process in step S50, where the newly calculated positionalrelationship information is a significantly different value incomparison with the positional relationship information up to thepresent, the position and attitude estimation unit 37 determines thatthe calculated positional relationship information is of values with lowreliability including some error factor, and therefore does not updatethe positional relationship information. Note that the process in stepS50 may be omitted and the information stored in the storage unit 38 maybe updated with the newly calculated positional relationship informationall the time.

The calibration process at shipment and the calibration process duringoperation are executed as above.

Note that although the above-described calibration process at shipmentand calibration process during operation are the examples of performingthe process for calculating calibration data indicating a positionalrelationship between the millimeter-wave radar 11 and the stereo camera12 just one time, the process may be performed multiple times and theaverage value thereof may be stored in the storage unit 38 as the finalcalibration data. Further, in the case of using pieces of calibrationdata calculated multiple times, it is possible to calculate the finalcalibration data after excluding data largely deviated from the otherpieces of calibration data among the pieces of calibration datacalculated multiple times.

5. Examples of Targets in Calibration Process at Shipment

FIG. 25 illustrates other examples of the targets that can be used inthe calibration process at shipment.

In the calibration process at shipment, a ball 161 or a corner reflector162 that reflects a millimeter wave can be used as a target, asillustrated in FIG. 25 , for example.

To increase the calibration precision, it is desirable that the targetdetection unit 33, which detects the position of the target on the basisof the stereo image, detect the position of the target at the pixellevel. In the example of the target 51 illustrated in FIG. 2 , theintersection point 52 of the texture is calculated. In a case where thetarget is the ball 161, it is possible to detect the ball 161 in thestereo image as a circle by pattern matching, shape recognition for acircular shape or the like, and output the center position of thedetected ball 161 as the target detection position.

Further, where the target is the corner reflector 162, it is possible todetect the corner reflector 162 in the stereo image by performingpattern matching using a registered pattern for the corner reflector162, which is preliminarily registered, and output the center positionof the registered pattern for the detected corner reflector 162 as thetarget detection position.

FIG. 26 illustrates examples of the reflected signals of themillimeter-wave radar 11 as well as the disparity image that the stereocamera 12 calculates from the stereo image where the targets are balls161.

6. Examples of Targets in Calibration Process During Operation

Next, other examples of the targets that can be used in the calibrationprocess during operation will be described.

Examples of the targets that can be used in the calibration processduring operation include a general object existing in a trafficenvironment such as, for example, a part of another vehicle (forexample, a license plate), a part of the own vehicle, an advertisingdisplay, a sign, a traffic light, or a utility pole, in addition to apedestrian (human) described above.

A method of detecting a target position where the target is a pole-likeobject such as a utility pole will be described with reference to FIG.27 .

First, the target detection unit 33 detects vertical parallel lines froma stereo image and calculates an area surrounded by the detectedparallel lines as a pole area. In the example of FIG. 27 , one pole areais detected in the stereo image, but there are also cases where aplurality of pole areas is detected.

Then, the target detection unit 33 individually performs poledetermination on the detected pole area in the horizontal direction andthe vertical direction on the basis of the disparity information of thestereo image calculated by the disparity estimation unit 34.

Specifically, as illustrated in FIG. 27 , when the disparity viewed inthe horizontal direction transitions in the pole area such that thedisparity draws a convex curve extending upward, the target detectionunit 33 determines that the pole determination result in the horizontaldirection is true for the pole area, and when not, false.

Further, when the change in the disparity viewed in the verticaldirection is small (equal to or less than a predetermined value), thetarget detection unit 33 determines that the pole determination resultin the vertical direction is true for that pole area, and when not,false.

Lastly, the target detection unit 33 performs an AND operation on thepole determination result in the horizontal direction and the poledetermination result in the vertical direction. That is, when bothdetermination results, that is, the pole determination result in thehorizontal direction and the pole determination result in the verticaldirection are true, the target detection unit 33 outputs the position ofthe pole area as the target detection position.

As described above, when the target is a pole-like object such as autility pole or the like, the target detection unit 33 can detect thetarget by also using the disparity information calculated by thedisparity estimation unit 34 and output the target detection position.

The object detection system 1 described above can detect and calibratethe positional relationship information of the millimeter-wave radar 11and the stereo camera 12 at the pixel-level precision of the stereoimage, thereby achieving calibration with high precision.

Further, since the calibration during operation is executed using ageneral object existing in a traffic environment such as a pole-likeobject including a sign, a utility pole or the like, or a pedestrian,correction can be made (automatically) at any time even when thepositional relationship between the millimeter-wave radar 11 and thestereo camera 12 changes due to factors such as aging variation,vibration, and heat.

Although a description has been given of the example where the objectdetection system 1 is mounted in a vehicle in the above-describedexample, the present technology can also be mounted in other movingobjects that move on the land such as, for example, robots, in additionto vehicles.

Further, in the above-described example, the millimeter-wave radar 11 isemployed as a first sensor for calculating the three-dimensionalposition of a target on the first coordinate system, and the stereocamera 12 is employed as a second sensor for calculating thethree-dimensional position of a target on the second coordinate system.As the first sensor, in addition to the millimeter-wave radar 11,another radar-type sensor such as a radar using ultrasonic waves, alaser radar such as infrared rays, or a lidar may be employed. In otherwords, the first sensor may be any sensor as long as the sensor canobtain the position information of at least one of the lateral direction(horizontal direction) and longitudinal direction (vertical direction)and the position information of the depth direction.

7. Exemplary Computer Configuration

The series of processes including the above-described calibrationprocess can be executed by hardware or software. In a case where theseries of processes are to be executed by software, a programconstituting the software is installed in a computer. Here, examples ofthe computer include a computer incorporated into dedicated hardware anda general-purpose personal computer capable of executing various kindsof functions with various kinds of programs installed therein.

FIG. 28 is a block diagram illustrating an exemplary hardwareconfiguration of the computer in which the program executes the seriesof processes described above.

In the computer, a CPU (Central Processing Unit) 201, an ROM (Read OnlyMemory) 202, and an RAM (Random Access Memory) 203 are connected to eachother through a bus 204.

Moreover, an input/output interface 205 is connected to the bus 204. Aninput unit 206, an output unit 207, a storage unit 208, a communicationunit 209, and a drive 210 are connected to the input/output interface205.

The input unit 206 includes a keyboard, a mouse, a microphone and thelike. The output unit 207 includes a display, a speaker and the like.The storage unit 208 includes a hard disk, non-volatile memory and thelike. The communication unit 209 includes a network interface and thelike. The drive 210 drives a removable recording medium 211 such as amagnetic disk, an optical disk, a magneto-optical disk, or asemiconductor memory.

In the computer configured as above, for example, the CPU 201 loads theprogram stored in the storage unit 208 into the RAM 203 via theinput/output interface 205 and the bus 204 and executes the program,through which the above-described series of processes are performed.

In the computer, the program can be installed in the storage unit 208via the input/output interface 205 by attaching the removable recordingmedium 211 to the drive 210. Further, the program can be received by thecommunication unit 209 via a wired or wireless transmission medium suchas a local area network, the Internet, or digital satellitebroadcasting, and installed into the storage unit 208. Additionally, theprogram can be installed in advance in the ROM 202 or the storage unit208.

Note that the program executed by the computer may be a program thatperforms the processes in a chronological order in the order describedin the present specification, or may be a program that performs theprocesses in parallel or at necessary timing on occasions of calls, forexample.

The steps described in the flowcharts may not only be performed inchronological order in the described order, but also be executed inparallel or at necessary timing on occasions of calls, for example, andthe steps are not necessarily processed in chronological order.

In the present specification, the system refers to a collection of aplurality of components (apparatuses, modules (parts) and the like), andit does not matter whether or not all the components are within the samehousing. Therefore, a plurality of apparatuses housed in separatehousings and connected via a network, and a single apparatus in which aplurality of modules is housed within a single housing is, in eithercase, the system.

The embodiments of the present technology are not limited to theabove-described embodiments, and various modifications can be madewithout departing from the gist of the present technology.

For example, a mode of combining all or part of the plurality ofembodiments described above can be employed.

For example, the present technology can take a configuration of cloudcomputing in which one function is shared and processed in cooperationby a plurality of apparatuses through a network.

Further, each of the steps described in the above-described flowchartscan be executed by a single apparatus, but can also be shared andexecuted by a plurality of apparatuses.

Moreover, in a case where a plurality of processes is included in asingle step, the plurality of processes included in the single step canbe executed by a single apparatus, or can also be shared and executed bya plurality of apparatuses.

Note that the effects described in the present specification are merelyillustrative and not limited, and there may be effects other than thosedescribed in the present specification.

Note that the present technology can also be configured as below.

(1)

A signal processing apparatus including:

a first position calculation unit that calculates a three-dimensionalposition of a target on a first coordinate system from a stereo imagecaptured by a stereo camera;

a second position calculation unit that calculates a three-dimensionalposition of the target on a second coordinate system from a sensorsignal of a sensor capable of obtaining position information of at leastone of a lateral direction and a longitudinal direction and positioninformation of a depth direction;

a correspondence detection unit that detects a correspondencerelationship between the target on the first coordinate system and thetarget on the second coordinate system; and a positional relationshipinformation estimating unit that estimates positional relationshipinformation of the first coordinate system and the second coordinatesystem on the basis of the detected correspondence relationship.

(2)

The signal processing apparatus according to (1),

in which the correspondence detection unit detects the correspondencerelationship after collating individual targets on the first coordinatesystem and the second coordinate system with prior arrangementinformation of the target and identifying the target.

(3)

The signal processing apparatus according to (1) or (2),

in which the correspondence detection unit detects the correspondencerelationship by superimposing the three-dimensional position of thetarget on the first coordinate system and the three-dimensional positionof the target on the second coordinate system over each other and makingthe targets arranged closest to each other correspond to each other.

(4)

The signal processing apparatus according to any one of (1) to (3),

in which the first position calculation unit calculates thethree-dimensional position of the target whose motion is detected, and

the second position calculation unit calculates the three-dimensionalposition of the target whose motion is detected.

(5)

The signal processing apparatus according to any one of (1) to (4),

in which the first position calculation unit calculatesthree-dimensional positions of a plurality of the targets from at leastone stereo image in one or more frames,

the second position calculation unit calculates three-dimensionalpositions of the plurality of the targets from sensor signals in one ormore frames, and

the correspondence detection unit detects correspondence relationshipsbetween the plurality of the targets.

(6)

The signal processing apparatus according to (5),

in which the first position calculation unit calculatesthree-dimensional positions of a plurality of the targets from a stereoimage in one frame, and

the second position calculation unit calculates three-dimensionalpositions of the plurality of the targets from sensor signals in oneframe.

(7)

The signal processing apparatus according to any one of (1) to (6),further including:

a storage unit that stores the three-dimensional positions of the targetcalculated by the first position calculation unit and the secondposition calculation unit,

in which, where a predetermined number or more of three-dimensionalpositions of the target are accumulated in the storage unit, thecorrespondence detection unit starts detection of the correspondencerelationship.

(8)

The signal processing apparatus according to any one of (1) to (7),

in which a plurality of the targets is arranged at different positionsin the depth direction.

(9)

The signal processing apparatus according to any one of (1) to (8),

in which a plurality of the targets is arranged at different positionsin the lateral direction.

(10)

The signal processing apparatus according to any one of (1) to (9),

in which a plurality of the targets is arranged at an identical heightposition.

(11)

The signal processing apparatus according to any one of (1) to (10),

in which a plurality of the targets is arranged at positions that arenot overlapped with each other when viewed from the stereo camera.

(12)

The signal processing apparatus according to any one of (1) to (10),

in which the target is a human.

(13)

The signal processing apparatus according to any one of (1) to (10),

in which the target is an object with a predetermined texture.

(14)

The signal processing apparatus according to any one of (1) to (10),

in which the target is a pole-like object.

(15)

The signal processing apparatus according to any one of (1) to (14),

in which the positional relationship information of the first coordinatesystem and the second coordinate system is a rotation matrix and atranslation vector.

(16)

The signal processing apparatus according to any one of (1) to (15),

in which the sensor is a millimeter-wave radar.

(17)

A signal processing method including the steps of:

calculating a three-dimensional position of a target on a firstcoordinate system from a stereo image captured by a stereo camera;

calculating a three-dimensional position of the target on a secondcoordinate system from a sensor signal of a sensor capable of obtainingposition information of at least one of a lateral direction and alongitudinal direction and position information of a depth direction;

detecting a correspondence relationship between the target on the firstcoordinate system and the target on the second coordinate system; and

estimating positional relationship information of the first coordinatesystem and the second coordinate system on the basis of the detectedcorrespondence relationship.

(18)

A program for causing a computer to execute a process including thesteps of:

calculating a three-dimensional position of a target on a firstcoordinate system from a stereo image captured by a stereo camera;

calculating a three-dimensional position of the target on a secondcoordinate system from a sensor signal of a sensor capable of obtainingposition information of at least one of a lateral direction and alongitudinal direction and position information of a depth direction;

detecting a correspondence relationship between the target on the firstcoordinate system and the target on the second coordinate system; and

estimating positional relationship information of the first coordinatesystem and the second coordinate system on the basis of the detectedcorrespondence relationship.

REFERENCE SIGNS LIST

1 Object detection system, 11 Millimeter-wave radar, 12 Stereo camera,13 Signal processing apparatus, 21L Left camera, 21R Right camera, 31Target detection unit, 32 Three-dimensional position calculating unit,33 Target detection unit, 34 Disparity estimation unit, 35Three-dimensional position calculating unit, 36 Correspondence detectionunit, 37 Position and attitude estimation unit, 38 Storage unit, 51Target, 71 Motion detection unit, 72 Peak detection unit, 73 ANDoperation unit, 81 Motion area detection unit, 82 Image recognitionunit, 83 AND operation unit, 84 Center position calculation unit, 201CPU, 202 ROM, 203 RAM, 206 Input unit, 207 Output unit, 208 Storageunit, 209 Communication unit, 210 Drive

The invention claimed is:
 1. A signal processing apparatus comprising:circuitry configured to: calculate a three-dimensional position of atarget on a first coordinate system from first data obtained by a firstsensor; calculate a three-dimensional position of the target on a secondcoordinate system from second data obtained by a second sensor otherthan the first sensor; detect a correspondence relationship between thetarget on the first coordinate system and the target on the secondcoordinate system; and estimate positional relationship information ofthe first coordinate system and the second coordinate system on thebasis of the detected correspondence relationship.
 2. The signalprocessing apparatus according to claim 1, wherein the circuitry isconfigured to detect the correspondence relationship after collatingindividual targets on the first coordinate system and the secondcoordinate system with prior arrangement information of the target andidentifying the target.
 3. The signal processing apparatus according toclaim 1, wherein the circuitry is configured to detect thecorrespondence relationship by superimposing the three-dimensionalposition of the target on the first coordinate system and thethree-dimensional position of the target on the second coordinate systemover each other and making the targets arranged closest to each othercorrespond to each other.
 4. The signal processing apparatus accordingto claim 1, wherein the circuitry is configured to calculate thethree-dimensional position of the target whose motion is detected, andthe circuitry is configured to calculate the three-dimensional positionof the target whose motion is detected.
 5. The signal processingapparatus according to claim 1, wherein the circuitry is configured tocalculate three-dimensional positions of a plurality of the targets fromat least first data at one or more times, the circuitry is configured tocalculate three-dimensional positions of the plurality of the targetsfrom second data at one or more times, and the circuitry is configuredto detect correspondence relationships between the plurality of thetargets.
 6. The signal processing apparatus according to claim 5,wherein the circuitry is configured to calculate three-dimensionalpositions of a plurality of the targets from first data, and thecircuitry is configured to calculate three-dimensional positions of theplurality of the targets from second data.
 7. The signal processingapparatus according to claim 1, further comprising: a memory that storesthe three-dimensional positions of the target calculated by thecircuitry, wherein, where a predetermined number or more ofthree-dimensional positions of the target are accumulated in the memory,the circuitry starts detection of the correspondence relationship. 8.The signal processing apparatus according to claim 1, wherein aplurality of the targets is arranged at different positions in a depthdirection.
 9. The signal processing apparatus according to claim 1,wherein a plurality of the targets is arranged at different positions ina lateral direction.
 10. The signal processing apparatus according toclaim 1, wherein a plurality of the targets is arranged at an identicalheight position.
 11. The signal processing apparatus according to claim1, wherein a plurality of the targets is arranged at positions that arenot overlapped with each other when viewed from the first sensor. 12.The signal processing apparatus according to claim 1, wherein the targetis a human.
 13. The signal processing apparatus according to claim 1,wherein the target is an object with a predetermined texture.
 14. Thesignal processing apparatus according to claim 1, wherein the target isa pole-like object.
 15. The signal processing apparatus according toclaim 1, wherein the positional relationship information of the firstcoordinate system and the second coordinate system is a rotation matrixand a translation vector.
 16. The signal processing apparatus accordingto claim 1, wherein the second sensor is a millimeter-wave radar.
 17. Asignal processing method comprising the steps of: using at least oneprocessor: calculating a three-dimensional position of a target on afirst coordinate system from first data obtained by a first sensor;calculating a three-dimensional position of the target on a secondcoordinate system from second data obtained by a second sensor otherthan the first sensor; detecting a correspondence relationship betweenthe target on the first coordinate system and the target on the secondcoordinate system; and estimating positional relationship information ofthe first coordinate system and the second coordinate system on thebasis of the detected correspondence relationship.
 18. At least onenon-transitory computer-readable storage medium encoded with executableinstructions that, when executed by at least one processor, cause the atleast one processor to perform a method comprising: calculating athree-dimensional position of a target on a first coordinate system fromfirst data obtained by a first sensor; calculating a three-dimensionalposition of the target on a second coordinate system from second dataobtained by a second sensor other than the first sensor; detecting acorrespondence relationship between the target on the first coordinatesystem and the target on the second coordinate system; and estimatingpositional relationship information of the first coordinate system andthe second coordinate system on the basis of the detected correspondencerelationship.